Information processing method and apparatus

ABSTRACT

It is intended to solve the problem that, when an operator performs an operation using two-dimensional input means while seeing a picture frame seen from a viewpoint of the operator, an input operation is difficult to perform when the direction of the line of sight of the operator is unstable, since a designated direction greatly deviates in a world coordinate system. For that purpose, viewpoint information is detected, and an instruction of the operator for operating a position of a pointer image is input. A designated direction in a pointer coordinate system is obtained in accordance with the operator&#39;s instruction, and the pointer image is generated based on the designated direction. The pointer coordinate system is changed from the detected viewpoint information in accordance with a specific instruction of the operator.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to an information processing methodand apparatus for displaying an image in which a pointer image operatedby an operator is synthesized.

[0003] 2. Description of the Related Art

[0004] In virtual reality (VR), it is possible to see athree-dimensional virtual space from a viewpoint of an operator using adisplay, such as an HMD (head mounted display) or the like, and performan operation in such an environment. In mixed reality (MR), an operatorcan see a real space and three-dimensional virtual space in a superposedstate using a see-through HMD.

[0005] In a telepresence system, an operator can perform an operation asif he is actually present at a remote location, by operating a cameraprovided at the remote location and performing the operation whileseeing an image taken by the camera.

[0006] In such VR, MR or a telepresence system, an operator sometimesperforms an operation for a real or virtual three-dimensional space. Forexample, the operator moves a virtual object within a three-dimensionalvirtual space, or designates an object in a real thee-dimensional spaceand displays information relating to the object on a display.Conventionally, such an operation is performed using a three-dimensionalinput device, such as a magnetic sensor, an optical sensor or the like,that can perform three-dimensional designation.

[0007] In contract to methods for inputting an operation for athree-dimensional space using a three-dimensional input device, methodsfor performing an operation in a three-dimensional space using atwo-dimensional input device, such as a mouse or the like, have beenknown. As an example of such methods, a method for performing input to athree-dimensional space by obtaining a z-coordinate value in thedirection of the depth from a designated position on screen coordinates(x, y) obtained by perspective transformation of a three-dimensionalspace from a previously prepared table, using an input device, such as amouse or the like.

[0008] In any of such methods using a two-dimensional input device, aninput operation is performed by operating a pointer by a two-dimensionalinput device in an image obtained by perspective transformation of athee-dimensional space. An example of VR application software forinputting an operation from a two-dimensional input device for athree-dimensional space in such a conventional input method will now bedescribed.

[0009] The operator mounts an HMD 201 shown in FIG. 1. The HMD 201provides the operator with a three-dimensional virtual space. The HMD201 includes a head-position/posture sensor 202 that outputs theposition and the posture of the HMD 201. The operator has atwo-dimensional input device 101 shown in FIG. 2 in his hand. Theoperator can input two-dimensional values using a two-dimensionalpointing stick 102. That is, as shown in FIG. 2, the operator can inputvalues (x, y )=(0, 0)-(100, 100) depending on the direction and theangle of inclination of the pointing stick 102. The two-dimensionalinput device 101 also includes a first button 103 and a second button104.

[0010]FIG. 3 illustrates an outline of a three-dimensional virtual spacepresented to the operator using such application software. Athree-dimensional virtual space 300 comprises a rectangularparallelepiped 301. Within the rectangular parallelepiped 301, virtualspheres A 302, B 303 and C 304 are floating. The operator is alsopresent within the rectangular parallelepiped 301. Reference numeral 306represents a world coordinate system whose origin is at a corner of therectangular parallelepiped 301. FIG. 4 illustrates how the virtual space300 is seen from the operator's viewpoint.

[0011]FIG. 5 represents a picture frame 500 displayed in the operator'sHMD 201. The picture frame 500 has a width comprising 640 pixels and aheight comprising 480 pixels. A coordinate position on the picture frame500 is represented, for example, as (x, y)=(640, 480) making the upperleft corner of the picture frame 500 the origin. A pointer 501 isdisplayed on the picture frame 500, and moves by the user's operation ofthe two-dimensional pointing stick 102. The pointer 501 moves on thepicture frame 500 in accordance with the amount of operation of thepointing stick 102. In FIG. 5, the pointer 501 assumes a position of (x,y)=(490, 80).

[0012] When the operator depresses the first button 103, an objectpresent under the pointer 501 is designated. For example, in FIG. 5, thepointer 501 is displayed in a state of being superposed with the virtualsphere A 302. Hence, when the operator depresses the first button 103,the virtual sphere A 302 is designated. At that time, determinationwhether or not the pointer 501 is present on the virtual sphere A 302can be performed, for example, by acquiring the z buffer value of thecoordinates (490, 80).

[0013] The above-described three-dimensional input device using amagnetic sensor or an optical sensor is generally expensive.Furthermore, since the number of objects to be measured at a time issometimes limited depending on the sensor being used, it is sometimesimpossible to measure the input device using a sensor. In addition, forexample, in a three-dimensional input device using an optical sensor,measurement cannot be performed if a body or an object is presentbetween a measuring apparatus and an object to be measured.

[0014] The above-described problem is solved by using a two-dimensionalinput device, such as a mouse or the like. However, in the conventionalmethod of moving a pointer using a two-dimensional input device in animage obtained by performing perspective transformation of athree-dimensional space, if the direction of the operator's line ofsight is unstable, the designated direction of the pointer greatlychanges in the three-dimensional space. In other words, since thedesignated direction is defined by a field-of-view coordinate system,the designated direction greatly changes in a world coordinate systemwhen the field-of-view coordinate system is not fixed.

[0015] For example, when the operator mounts an HMD, the designateddirection greatly changes even if the position coordinates of thepointer do not change on the picture frame, if the operator's headfluctuates due to the weight of the HMD. Accordingly, in order toexactly designate an object in a three-dimensional space, the operatormust fix his head, resulting in pain for the operator. This problem willnow be described with reference to drawings.

[0016]FIG. 6 is a diagram illustrating the relationship between a worldcoordinate system 306 and a field-of-view coordinate system 601. Theposition and the posture of the field-of-view coordinate system 601 inthe world coordinate system 306 are obtained from the headposition/posture sensor 202. An image plane 602 is arranged relative tothe field-of-view coordinate system 601. A designated direction 603 ofthe operator is represented by an arrow connecting the origin of thefield-of-view coordinate system 601 and the pointer 501.

[0017]FIG. 7 is a diagram illustrating how the operator's designateddirection 603 changes when the direction of the operator's line of sightchanges due to fluctuation of the operator's head. As shown in FIG. 7,when the operator's line of sight changes, the relationship between theworld coordinate system 306 and the line-of-sight coordinate system 601changes, to change the positions of the image plane 602 and the pointer501, thereby greatly changing the designated direction 603 in the worldcoordinate system 306.

[0018]FIG. 8 illustrates a picture frame 801 displayed in the operator'sHMD 201. A frame 802 indicated by broken lines indicates the position ofthe picture frame before the line of sight changes, i.e., at the time ofthe state shown in FIG. 6. As can be understood from FIG. 8, before theline of sight changes, the pointer 501 is displayed in a state of beingsuperposed with the virtual sphere A 302. The position of the pointer501 changes due to a change in the direction of the line of sight, andis displayed in a state of being separated from the virtual sphere A302. Accordingly, in order to designate the virtual sphere A 302, it isnecessary to again move the pointer 501. The same operation must beperformed if the direction of the line of sight changes due tofluctuation of the head after the movement. That is, in order to performa stable operation, it is necessary to fix the operator's head mountingthe HMD 201, resulting in a pain for the operator.

[0019] Although the above-described example represents a case of VRapplication software using an HMD, similar problems will arise in MRapplication software or telepresence application software. For example,in a telepresence system, when a camera is placed at a remote location,the photographing direction of the camera sometimes fluctuates due tomechanical factors or a change in the external environment caused bywind or the like. The above-described conventional approach cannot dealwith such a problem because it presumes a state in which the position ofthe viewpoint and the direction of the line of sight are fixed.

SUMMARY OF THE INVENTION

[0020] It is an object of the present invention to solve theabove-described problems.

[0021] It is another object of the present invention to allow to performa stable input operation even if the direction of an operator's line ofsight is unstable.

[0022] According to one aspect of the present invention, an informationprocessing method for displaying an image in which a pointer image issynthesized includes the steps of detecting viewpoint information,inputting an instruction of an operator for operating a position of thepointer image, obtaining a designated direction in a pointer coordinatesystem in accordance with the operator's instruction, and generating thepointer image based on the designated direction. The pointer coordinatesystem is changed from the detected viewpoint information in accordancewith a specific instruction of the operator.

[0023] According to another aspect of the present invention, aninformation processing apparatus includes a sensor adapted to detectviewpoint information of an operator, an operation unit adapted to inputan instruction of the user for operating a position of a pointer image,a designated-direction determination unit adapted to obtain a designateddirection in a pointer coordinate system in accordance with theinstruction input to the operation unit, an image generation unitadapted to generate the pointer image based on the designated directionand synthesizing the pointer image with another image, and a displayunit adapted to display a synthetic image generated by the imagegeneration unit. The pointer coordinate system is changed from thedetected viewpoint information in accordance with a specific instructionof the operator.

[0024] According to still another aspect of the present invention, in aprogram for realizing an information processing method for displaying animage in which a pointer is synthesized, the method includes the stepsof detecting viewpoint information, inputting an instruction of anoperator for operating a position of the pointer image, obtaining adesignated direction in a pointer coordinate system in accordance withthe operator's instruction, and generating the pointer image based onthe designated direction. The program includes a program for changingthe pointer coordinate system from the detected viewpoint information inaccordance with a specific instruction of the operator.

[0025] Other features and advantages of the present invention will beapparent from the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

[0026]FIG. 1 is a diagram illustrating an HMD mounted on an operator;

[0027]FIG. 2 is a diagram illustrating a two-dimensional input deviceheld by a hand of the operator;

[0028]FIG. 3 is a diagram illustrating a three-dimensional virtual spacepresented to the operator;

[0029]FIG. 4 is a diagram illustrating the three-dimensional virtualspace as seen from a viewpoint of the operator;

[0030]FIG. 5 is a diagram illustrating a picture frame displayed in theHMD of the operator;

[0031]FIG. 6 is a diagram illustrating the relationship between a worldcoordinate system and a field-of-view coordinate system;

[0032]FIG. 7 is a diagram illustrating how a designated direction of theoperator changes when the direction of the line of sight of the operatorchanges, in a conventional input method;

[0033]FIG. 8 is a diagram illustrating how the picture frame shown inthe HMD of the operator changes before and after the direction of theline of sight of the operator changes, in the conventional input method;

[0034]FIG. 9 is a block diagram illustrating an example of theconfiguration of an input method according to an embodiment of thepresent invention;

[0035]FIG. 10 is a flowchart illustrating a processing procedure of theinput method of the embodiment;

[0036]FIG. 11 is a diagram illustrating resetting of the designateddirection in the input method of the embodiment;

[0037]FIG. 12 is a diagram illustrating a pointer coordinate system whenthe position of a viewpoint is subjected to parallel movement, in theinput method of the embodiment;

[0038]FIG. 13 is a diagram illustrating the pointer coordinate systemwhen the direction of the line of sight changes, in the input method ofthe embodiment;

[0039]FIG. 14 is a diagram illustrating changes in the operationdirection of a two-dimensional pointing stick and a direction vector, inthe input method of the embodiment;

[0040]FIG. 15 is a diagram illustrating a state in which a virtualsphere is designated, in the input method of the embodiment;

[0041]FIG. 16 is a diagram illustrating that the designated directiondoes not change even if the direction of the line of sight of theoperator changes, in the input method of the embodiment;

[0042]FIG. 17 is a diagram illustrating how the picture frame displayedin the HMD of the operator changes before and after the direction of theline of sight of the operator changes, in the input method of theembodiment;

[0043]FIG. 18 is a diagram illustrating restoration of the designateddirection when a pointer becomes off the field of view; and

[0044]FIG. 19 is a diagram when a direction vector is represented bypolar coordinates, in the input method of the embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] A preferred embodiment of the present invention will now bedescribed with reference to the drawings. In this embodiment, theabove-described VR application software in which the operator mounts anHMD and designates a virtual sphere present in a virtualthree-dimensional space is used. However, an input method different fromthe above-described method is adopted. The input method of thisembodiment can be applied to arbitrary VR or MR application software, ortelepresence application software.

[0046]FIG. 9 is a block diagram illustrating an example of theconfiguration of the input method of the embodiment. In FIG. 9, thereare shown a two-dimensional input device 101, a head position/posturesensor 202, an HMD 201, designated-direction determination means 901,designated-direction restoration means 902, designated-directionresetting means 903, and image generation means 904. Each of these unitswill now be described.

[0047] The two-dimensional input device 101 comprises, for example, apointing stick. The operator can input two-dimensional values byoperating the two-dimensional input device 101. Any other device thatcan input two-dimensional values, such as a mouse or the like, may alsobe used as the two-dimensional input device 101. The two-dimensionalinput device 101 may also comprise, for example, four buttons, i.e.,upper and lower buttons, and left and right buttons. Two-dimensionalvalues that are output are transmitted to the designated-directiondetermination means 901.

[0048] The head position/posture sensor 202 comprises, for example, amagnetic sensor mounted in the HMD 201, and outputs six degrees offreedom, i.e., the position (three degrees of freedom) and the posture(three degrees of freedom) of the viewpoint of the HMD 201, whenevernecessary. The head position/posture sensor 202 is not always requiredto output six degrees of freedom. For example, when the position of theHMD 201 is fixed, the head position/posture sensor 202 is required tooutput only three degrees of freedom relating to the posture. Any otherdevice that can output information relating to the position and theposture of the HMD 201, such as a rotary encoder or an optical sensor,may also be used as the head position/posture sensor 202. Theposition/posture information that is output is transmitted to thedesignated-direction determination means 901.

[0049] The HMD 201 presents an image from a viewpoint of the operator tothe operator. The designated-direction determination means 901 receivestwo-dimensional values and the position/posture information of the HMD201 from the two-dimensional input device 101 and the headposition/posture sensor 202, respectively, and determines a designateddirection. The designated-direction resetting means 903 resets thedesignated direction. The designated-direction restoration means 902restores the designated direction. The image generation means 904receives the designated-direction information and the position/postureinformation of the HMD 201 from the designated-direction determinationmeans 901, draws a three-dimensional virtual space that can beaccommodated within the field of view of the HMD 201, also draws apointer representing the designated direction, and transmits the drawnimage to the HMD 201.

[0050]FIG. 10 is a flowchart illustrating a processing procedure of theinput method of the embodiment. The input method of the embodiment willnow be sequentially described with reference to FIG. 10.

[0051] In step S101, the position and the posture of the HMD 201 areacquired from the head position/posture sensor 202. The acquiredposition/posture information is transmitted to the designated-directiondetermination means 901. In step S102, it is determined whether or notthe second button 104 is depressed. If the result of the determinationin step S102 is affirmative, the process proceeds to step S103. If theresult of the determination in step S102 is negative, the processproceeds to step S104. In step S103, the designated direction is resetby the designated-direction resetting means 903.

[0052]FIG. 11 is a diagram illustrating resetting of the designateddirection. When the second button 104 has been depressed, thedesignated-direction resetting means 903 generates a pointer coordinatesystem 1101 based on the position/posture information of the HMD 201obtained from the head position/posture sensor 202. In the pointercoordinate system 1101, a viewpoint position 1102 of the HMD 201 is madethe origin, and the forward direction, the downward direction and therightward direction are made the z axis, the y axis and the x axis,respectively. It is assumed that in the pointer coordinate system 1101,the position of the origin moves so as to be linked with the viewpointposition 1102, and the posture is always constant with respect to aworld coordinate system 306. Accordingly, when the viewpoint position1102 performs parallel movement as shown in FIG. 12, the pointercoordinate system 1101 moves together. However, when the direction ofthe line of sight changes as shown in FIG. 13, the pointer coordinatesystem 1101 does not change.

[0053] When the pointer coordinate system 1101 is generated, thedesignated-direction resetting means 903 simultaneously generates adirection vector 1103 having a magnitude of 1. A pointer 501 is drawn atthe intersection of a straight line extended from the origin of thepointer coordinate system 1101 in the direction of the direction vector1103 and an image plane 602. When resetting the designated direction,the pointer 501 is always at the center of the image plane 602, based onthe definition of the pointer coordinate system 1101 and the directionvector 1103.

[0054] When the resetting of the designated direction in step S103 hasbeen terminated, then, in step S104, two-dimensional input values areacquired from the two-dimensional input device 101. In this step, theoperator can input two-dimensional values using the two-dimensionalpointing stick 102 shown in FIG. 2. That is, as shown in FIG. 2, values(x, y)=(0, 0)-(100, 100) can be input depending on the direction and theangle of inclination of the pointing stick 102. The two-dimensionalinput values are transmitted to the designated-direction determinationmeans 901.

[0055] In step S105, the designated direction 603 is determined by thedesignated-direction determination means 901. The designated direction603 is defined by a straight line extended from the origin of thepointer coordinate system 1101 in the direction of the direction vector1103. As shown in FIG. 14, the direction vector 1103 changes so as to godownward and rightward when the two-dimensional pointing stick 102 isinclined downward and rightward, respectively.

[0056] In order to cause the directional vector 1103 to change as shownin FIG. 14, the relationship between two-dimensional values and thedirection vector 1103 is defined, for example, in the following manner.As shown in FIG. 19, an angle made by the projection of the directionvector 1103 on the x-z plane of the pointer coordinate system 1101 withrespect to the z axis is represented by θ, and an angle made by thedirection vector 1103 having a magnitude of 1 with respect to the x-zplane of the pointer coordinate system 1101 is represented by Φ.Variations dθ and dΦ of θ and Φ, respectively, are defined as follows:

dθ=(x 1−50)/200π

dΦ=(y 1−50)/200π

[0057] where radians x1 and y1 are two-dimensional input values from thepointing stick.

[0058] By defining dθ and dΦ in the above-described manner andintegrating the variations, it is possible to obtain the values of θ andΦ at a certain time, and operate the direction vector 1103 as shown inFIG. 14 according to the above-described equations.

[0059] When representing the direction vector 1103 in an orthogonalcoordinate system using the values of θ and Φ, the following equationsare used:

x=sin θ·cos Φ

y=sin Φ

z=cos θ·cos Φ.

[0060] Although the method of representing the direction vector 1103using a polar coordinate system has been illustrated, any other methodmay also be adopted, provided that the direction of inclination of thepointing stick 102 coincides with the direction of change of thedirection vector 1103, as shown in FIG. 14. For example, the followingmethod may be adopted.

[0061] It is assumed that, when resetting the designated direction, thedirection vector 1103 is represented by (x, y, z)=(0, 0, 1) in thepointer coordinate system 1101, where x changes in a lateral directionof the pointing stick 102, y changes in a longitudinal direction of thepointing stick 102, and z is invariable. Variations dx and dy of x andy, respectively, are defined as follows:

dx=(x 1−50)/50

dy=(y 1−50)/50,

[0062] where x1 and y1 are two-dimensional input values of the pointingstick 102. By defining dx and dy in the above-described manner andintegrating the variations, it is possible to operate the directionvector 1103 as shown in FIG. 14 when the pointing stick 102 is moved.

[0063] The fact that, by defining the designated direction in theabove-described manner, a stable input operation can be performed evenif the direction of the line of sight of the operator is unstable willnow be described with reference to FIGS. 15 and 16. FIG. 15 illustratesa state in which, after resetting the designated direction, the virtualsphere A 302 is designated by operating the two-dimensional input device101. At that time, the direction vector 1103 is defined by the pointercoordinate system 1101, and the posture of the designated coordinatesystem 1101 is constant in the world coordinate system 306. Accordingly,even if the direction of the line of sight of the operator changes andthe image plane 602 moves in the world coordinate system 306 as shown inFIG. 16, the designated direction 603 is constant in the worldcoordinate system 306, and remains to designate the virtual sphere A302.

[0064]FIG. 17 represents a picture frame 1701 displayed in the HMD 201of the operator. A frame 1702 indicated by broken lines represents theposition of the picture frame before the line of sight changes, i.e., atthe time of the state shown in FIG. 15. As can be understood from FIG.17, even if the direction of the line of sight changes, the pointer 501remains to designate the virtual sphere A 302.

[0065] While in the conventional approach, as shown in FIGS. 7 and 8,the designated direction 603 greatly changes when the direction of theline of sight changes, it can be understood the method of the presentinvention can perform a stable input operation.

[0066] After determining the designated direction 603 in step S105 inthe above-described manner, then, in step S106, it is determined by thedesignated-direction restoration means 902 whether or not the pointer501 is within the field of view. This step is performed in order toprevent a case in which the operator loses sight of the designateddirection 603. If the result of the determination in step S106 isaffirmative, the process proceeds to step S108. If the result of thedetermination in step S106 is negative, the process proceeds to stepS107.

[0067] In step S107, the designated direction 603 is restored. Therestoration of the designated direction 603 is performed by moving thepointer coordinate system 1101 as shown in FIG. 18. As shown in FIG. 18,by setting the z axis of the designated coordinate system 1101 withinthe field of view to set the direction vector 1103 on the z axis, it ispossible to again accommodate the pointer 501 within the field of view.

[0068] In step S108, the image generation means 904 generates an imageto be presented to the operator. The image generation means 904generates a CG (computer graphics) image based on the position/postureinformation of the HMD 201 acquired in step S101, and also draws thepointer 501 at an intersection of the designated direction 603determined in step S105 or step S107 and the image plane 602.

[0069] As described above, according to the present invention, it ispossible to provide a method for performing a stable input operationusing two-dimensional input means, even if the direction of the line ofsight of the operator is unstable when the operator performs anoperation while seeing a picture frame seen from the operator'sviewpoint for a virtual or real three-dimensional space in VR or MRapplication software or telepresence application software.

[0070] (Other Embodiments)

[0071] The above-described embodiment uses VR application software inwhich only a three-dimensional virtual space is presented to theoperator. However, the present invention may also be applied to MRapplication software that presents not only a three-dimensional virtualspace but also a real three-dimensional space to the operator. The inputmethod of the present invention may also be applied to telepresenceapplication software.

[0072] Although in the above-described embodiment, the operator performsan operation of designating a virtual object present in athree-dimensional virtual space, the input method of the presentinvention may also be applied to an operation of designating a realobject in MR application software or telepresence application software.

[0073] Although in the above-described embodiment, an HMD is used as adevice for presenting a space to the operator, any other device may alsobe used provided that an image can be presented to the operator, such asa mere monitor. Such a configuration is adopted, for example, when theoperator sees an image taken by a robot camera from a remote location.

[0074] In the above-described embodiment, in step S103 of the processingprocedure, the pointer coordinate system 1101 is defined as a coordinatesystem in which the viewpoint position 1102 of the HMD 201 is made theorigin, and the forward direction, the downward direction and therightward direction are made the z axis, the y axis and the x axis,respectively. However, the pointer coordinate system 1101 is notnecessarily defined in the above-described manner, but may be defined inany other appropriate manner provided that the position is linked withthe viewpoint position 1102, and the posture is fixed in the worldcoordinate system 306.

[0075] Although in the above-described embodiment, in step S106 of theprocessing procedure, it is determined whether or not the pointer 501 iswithin the field of view, this step may be omitted. In such a case,although there exists a moment in which the pointer 501 is not withinthe field of view, this input method is more effective depending on thetype of application software.

[0076] Although in the above-described embodiment, a pointer is used inorder to represent the designated direction on a picture frame, anyother appropriate device may also be used provided that the designateddirection can be represented on a picture frame. For example, a virtualthree-dimensional object, such as a laser or the like, for representingthe designated direction may be used.

[0077] The objects of the present invention may also be achieved bysupplying a computer within an apparatus or a system connected tovarious devices so as to operate the various devices in order to realizethe functions of the above-described embodiments with program codes ofsoftware for realizing the functions of the above-described embodiments,and operating the various devices in accordance with a program stored inthe computer (or a CPU (central processing unit) or an MPU(microprocessor unit)) of the system or the apparatus.

[0078] In such a case, the program codes themselves of the softwarerealize the functions of the above-described embodiments, so that theprogram codes themselves, or means for supplying the computer with theprogram codes, such as a storage medium storing the program codes,constitutes the present invention.

[0079] For example, a flexible disk, a hard disk, an optical disk, amagnetooptical disk, a CD(compact disc)-ROM (read-only memory), amagnetic tape, a nonvolatile memory card, a ROM or the like may be usedas the storage medium storing the program codes.

[0080] Such program codes, of course, constitute the present inventionnot only when the functions of the above-described embodiments arerealized by executing supplied program codes by a computer, but alsowhen the functions of the above-described embodiments are realized bythe program codes in cooperation with an OS (operating system), otherapplication software or the like operating in the computer.

[0081] The present invention may, of course, also be applied to a casein which, after storing supplied program codes in a memory provided in afunction expanding board of a computer or in a function expanding unitconnected to the computer, a CPU or the like provided in the functionexpanding board or the function expanding unit performs a part or theentirety of actual processing, and the functions of the above-describedembodiments are realized by the processing.

[0082] The individual components shown in outline or designated byblocks in the drawings are all well known in the information processingmethod and apparatus arts and their specific construction and operationare not critical to the operation or the best mode for carrying out theinvention.

[0083] Although the present invention has been described above withrespect to the preferred embodiments, the invention is not limited tothe foregoing embodiments but many other modifications and variationsare possible within the spirit and scope of the appended claims of theinvention.

What is claimed is:
 1. An information processing method for displayingan image in which a pointer image is synthesized, said method comprisingthe steps of: detecting viewpoint information; inputting an instructionof an operator for operating a position of the pointer image; obtaininga designated direction in a pointer coordinate system in accordance withthe operator's instruction; and generating the pointer image based onthe designated direction, wherein the pointer coordinate system ischanged from the detected viewpoint information in accordance with aspecific instruction of the operator.
 2. An information processingmethod according to claim 1, wherein a virtual image is generated inaccordance with the viewpoint information, and wherein an image to bedisplayed is generated by synthesizing the pointer image with thevirtual image in accordance with the designated direction.
 3. Aninformation processing method according to claim 1, wherein the pointercoordinate system is a coordinate system that is generated based on theviewpoint information when the specific instruction is provided by theoperator, and whose posture is fixed in a world coordinate system.
 4. Aninformation processing method according to claim 1, wherein it isdetermined whether or not the designated direction is off a field ofview, and wherein if a result of the determination is affirmative, thedesignated direction is moved within the field of view.
 5. Aninformation processing apparatus comprising: a sensor adapted to detectviewpoint information of an operator; an operation unit adapted to inputan instruction of the operator for operating a position of a pointerimage; a designated-direction determination unit adapted to obtain adesignated direction in a pointer coordinate system in accordance withthe instruction input to said operation unit; an image generation unitadapted to generate the pointer image based on the designated directionand synthesizing the pointer image with another image; and a displayunit adapted to display a synthetic image generated by said imagegeneration unit, wherein the pointer coordinate system is changed usingthe detected viewpoint information in accordance with a specificinstruction of the operator.
 6. An information processing apparatusaccording to claim 5, wherein said display unit comprises a head mounteddisplay.
 7. An information processing apparatus according to claim 5,wherein said operation unit comprises two-dimensional input means.
 8. Aprogram for realizing an information processing method for displaying animage in which a pointer image is synthesized, said method comprisingthe steps of: detecting viewpoint information; inputting an instructionof an operator for operating a position of the pointer image; obtaininga designated direction in a pointer coordinate system in accordance withthe operator's instruction; and generating the pointer image based onthe designated direction, wherein said program comprises a program forchanging the pointer coordinate system from the detected viewpointinformation in accordance with a specific instruction of the operator.